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EXECUTIVE SUMMARY 


In response to the recession that began in 2007, the U.S. Congress passed, and President 
Barack Obama signed into law, the American Recovery and Reinvestment Act of 2009 (Pub. 

Law 1 1 1-5). At an estimated cost of $83 1 billion, this economic stimulus package sought to save 
and create jobs, provide temporary relief to those adversely affected by the recession, and invest 
in education, health, infrastructure, and renewable energy. States and school districts received 
$100 billion to secure teachers’ jobs and promote innovation in schools. This funding included 
$3 billion for School Improvement Grants (SIG), one of the Obama administration’s signature 
programs and one of the largest federal government investments in an education grant program. 
The SIG program awarded grants to states that agreed to implement one of four school 
intervention models — transformation, turnaround, restart, or closure — in their lowest-performing 
schools. Each of the models prescribed specific practices designed to improve student outcomes, 
including outcomes for high-need students such as English language learners (ELLs) (U.S. 
Department of Education 2010a). 

Given the importance of the SIG program and sizable investment in it, the Institute of 
Educations Sciences (IES) commissioned this evaluation to focus on four primary questions: 

• Did schools implementing a SIG-funded model use the improvement practices promoted by 
SIG, and how did that compare to use of those practices by schools not implementing a SIG- 
funded model? 

• Did use of SIG-promoted practices include a focus on ELLs, and did that focus on ELLs 
differ between schools implementing a SIG-funded model and schools not implementing 
one? 

• Did receipt of SIG funding to implement a school intervention model have an impact on 
outcomes for low-perfonning schools? 

• Was the type of school intervention model implemented related to improvement in outcomes 
for low-perfonning schools? 

The Every Student Succeeds Act of 2015 (ESSA) made changes to the SIG program that 
gives states and districts much more flexibility in determining how to turn around their lowest- 
achieving schools. For example, the U.S. Department of Education (ED) can no longer require 
the use of particular school intervention models, and funds previously set aside for SIG now flow 
through the regular Title I formula (Klein 2015). Despite these changes, findings on the first two 
questions remain useful to policymakers considering the future direction of funds for low- 
performing schools because they identify practice areas that these schools have and have not yet 
addressed. Further, findings on the first question provide a useful policy context for interpreting 
findings on the third question of whether the $3 billion federal investment had a positive impact 
on student achievement. For example, if use of the practices promoted by SIG was similar 
between schools that received grants and schools that did not, then it seems less likely that SIG 
would have a subsequent impact on student achievement. Findings on the fourth question, which 
shed light on whether certain models were associated with larger student achievement gains than 
other models, remain relevant for educators and administrators considering future evidence- 
based approaches for turning around low-perfonning schools. 
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This is the final report for this evaluation of SIG. Three earlier briefs focused on: (1) 
implementation of three interrelated levers for school improvement — granting low-performing 
schools operational authority, supporting them, and monitoring their progress (Herman et al. 
2014); (2) low-perfonning schools’ adoption of individual practices and combinations of 
practices promoted by SIG (Herrmann et al. 2015); and (3) states’ capacity to support school 
turnaround (Tanenbaum et al. 2015). An earlier report covered all major topic areas that SIG 
promoted, examining the extent to which schools implementing a SIG-fimded model and schools 
not implementing one reported using these practices in spring 2012, and whether use differed 
across these two groups of schools (Dragoset et al. 2015). This final report builds on the earlier 
briefs and report by including an additional year of data (spring 2013) and by examining whether 
receipt of SIG funding had an impact on student outcomes. 

Key findings 

SIG allowed grantees to implement one of four school intervention models (transformation, 
turnaround, restart, or closure). These models promoted the use of many improvement practices 
in four main areas: (1) adopting comprehensive instructional refonn strategies, (2) developing 
and increasing teacher and principal effectiveness, (3) increasing learning time and creating 
community-oriented schools, and (4) having operational flexibility and receiving support. It is 
worth knowing to what extent schools implementing these models with SIG funds (referred to as 
SIG-funded models throughout this report) actually used these practices, and how that compares 
to other schools. We examined the use of these SIG-promoted practices in two ways: (1) we 
conducted a descriptive analysis that compared use of these practices for 290 schools that 
implemented a SIG-funded model in 2012-2013 and 190 schools that did not, and (2) we used a 
regression discontinuity design (RDD) with data from 460 schools to examine whether 
implementation of a SIG-funded model in 2010-2011 had an impact on use of these practices. 

We also examined whether the SIG program had an impact on student outcomes. We used 
an RDD to calculate the overall impact of implementing any of the four SIG-funded models on 
test scores, high school graduation, and college enrollment. A sample of 190 schools eligible for 
SIG and 270 schools that were not eligible for SIG was used in the analysis. 

Finally, we examined whether certain intervention models were associated with larger 
student achievement gains than other models. We conducted a correlational analysis that 
examined the relationship between the type of model implemented and changes in student 
achievement over time. A sample of 270 schools that implemented a SIG-funded model in 2010- 
2011 was used in the analysis. 

Key findings included: 

• Although schools implementing SIG-funded models reported using more SIG- 

promoted practices than other schools, we found no evidence that SIG caused those 
schools to implement more practices. Our descriptive analysis found that schools 
implementing a SIG-fimded model used significantly more SIG-promoted practices than 
other schools (22.8 of the 35 practices examined [65 percent] versus 20.3 practices [58 
percent], a difference of 2.5 practices). Our more rigorous RDD analysis found a similar 
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difference of 3.3 practices, but it was not statistically significant. Therefore, we are unable to 
conclude that SIG caused the observed difference in use of practices. 

• Across all study schools, use of SIG-promoted practices was highest in comprehensive 
instructional reform strategies and lowest in operational flexibility and support. In the 

comprehensive instructional refonn strategies area, study schools reported using, on 
average, 7.1 of the 8 SIG-promoted practices examined (89 percent). In the operational 
flexibility and support area, study schools reported using, on average, 0.87 of the 2 SIG- 
promoted practices examined (43 percent). 

• There were no significant differences in use of English Language Learner (ELL)- 
focused practices between schools implementing a SIG-funded model and other 
schools. 

• Overall, across all grades, we found that implementing any SIG-funded model had no 
significant impacts on math or reading test scores, high school graduation, or college 
enrollment. 

• When we compared student achievement gains from different models in elementary 
grades (2nd through 5th), we found no evidence that one model was associated with 
larger gains than another. Lor higher grades (6th through 12th), the turnaround model 
was associated with larger student achievement gains in math than the transformation 
model. However, factors other than the SIG model implemented, such as baseline 
differences between schools implementing different models, may explain these differences 
in achievement gains. 

Background 

The SIG program aimed to support the implementation of school intervention models in 
low-perfonning schools. Although SIG was first authorized in 2001, this evaluation focused on 
SIG awards granted in 2010, when roughly $3.5 billion in SIG awards were made to 50 states 
and the District of Columbia, $3 billion of which came from the American Recovery and 
Reinvestment Act of 2009. States identified the low-performing schools eligible for SIG based 
on criteria specified by ED and then held competitions for local education agencies seeking 
funding to help turn around eligible schools. 

For the 2010 SIG competition, ED required states to categorize schools into three eligibility 
tiers based on the school’s level (elementary or secondary), Title I status, 1 and achievement or 
graduation rate. These tiers helped prioritize the distribution of SIG funds at the local level and 
determined the practices to be used for school turnaround. In general, SIG eligibility Tiers I and 
II included schools with the lowest achievement and most persistent achievement problems in 
each state. 


1 Title I, Part A (Title I) of the Elementary and Secondary Education Act provides financial assistance to local 
educational agencies and schools with many children from low-income families. A school receiving Title I funds 
that fails to meet adequate yearly progress targets can be assigned a Title I status of “in need of improvement, 
corrective action, or restructuring.” 
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ED required that each SIG-awarded school under Tier I or Tier II implement one of four 
school intervention models. These models required specific practices: 

• Transformation. This model required schools to replace the principal, adopt a teacher and 
principal evaluation system that accounted for student achievement growth as a significant 
factor, adopt a new governance structure, institute comprehensive instructional reforms, 
increase learning time, create community-oriented schools, and have operational flexibility. 

• Turnaround. This model required schools to replace the principal, replace at least 50 
percent of the school staff, institute comprehensive instructional reforms, increase learning 
time, create community-oriented schools, and have operational flexibility. 

• Restart. This model required schools to convert to a charter school or close and reopen 
under the management of a charter management organization or education management 
organization. 

• School closure. This model required districts to close schools and enroll their students in 
higher-achieving schools within the district. 

These required practices can be grouped into the four main topic areas promoted by SIG. 
Table ES. 1 lists these four broad areas and the objectives promoted by SIG within each. For 
example, replacing the principal falls under the “identifying and rewarding effective teachers and 
principals and removing ineffective ones” objective within the “developing and increasing 
teacher and principal effectiveness” topic area. The objectives listed in the table cover all 
practices promoted by SIG, which different models designated as either required or permissible. 
For example, adopting a teacher and principal evaluation system that accounted for student 
achievement growth was required under the transformation model but permissible under the 
turnaround and restart models. For detailed information about the practices that each model 
required, see Appendix F, Table F. 1 . 

We did not limit our examination to the specific practices required by each model. We 
instead focused on all required or permissible practices under the transformation or turnaround 
models because (a) both models prescribed a large set of overlapping practices, (b) restart model 
schools could choose to use any of those practices, and (c) an earlier report from this study 
(Herrmann et al. 2014) already presented findings on the implementation of required practices by 
schools using different models. 

Table ES.1. SIG objectives, by topic area 


Implementing comprehensive instructional reform strategies 


Using Data to Identify and Implement an Instructional Program 
Promoting the Continuous Use of Student Data 

Conducting Periodic Reviews to Ensure that the Curriculum is Being Implemented with Fidelity 

Implementing a New School Model (Such As an Academy with a Theme Focused on Science, Technology, 
Engineering and Math) 

Providing Supports and Professional Development to Staff to Assist ELL Students and Students with Disabilities 
Using and Integrating Technology-Based Supports 
Tailoring Strategies for Secondary Schools 
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Developing and increasing teacher and principal effectiveness 


Using Rigorous, Transparent, and Equitable Evaluation Systems 

Identifying and Rewarding Effective Teachers and Principals and Removing Ineffective Ones 
Providing High-Quality, Job-Embedded Professional Development or Supports 
Implementing Strategies to Recruit, Place, and Retain Staff 


Increasing learning time and creating community-oriented schools 


Using Schedules and Strategies That Provide Increased Learning Time or Increasing the Number of Hours per 
Year That School Was in Session 

Engaging Families and Communities and Providing a Safe School Environment That Meets Students’ Social, 
Emotional, and Health Needs 


Having operational flexibility and receiving support 


Having Primary Responsibility for Budget, Hiring, Discipline, or School Year Length Decisions 
Receiving Technical Assistance and Support 
Source: SIG application. 

ELL = English language learner. 

Research questions and study design 

Including the four primary research questions listed earlier (and italicized below), this report 
was guided by a total of seven research questions in three broad areas: 

Use of SIG-promoted practices 

1. Did schools implementing a SIG-funded model use the improvement practices promoted 
by SIG, and how did that compare to use of those practices by schools not implementing 
a SIG-funded model? 

2. Did receipt of SIG funding to implement a school intervention model have an impact on 
the number of SIG-promoted practices used by low-perfonning schools? 

3. Did schools’ use of SIG-promoted practices change over time? 

4. Did use of SIG-promoted practices include a focus on ELLs, and did that focus on ELLs 
differ between schools implementing a SIG-funded model and schools not implementing 
one? 

5. Did use of these ELL-focused improvement practices differ based on the prevalence of 
ELL students in the school or the achievement gap between ELL and other students? 

Whether SIG-funded intervention models improved student outcomes 

6. Did receipt of SIG funding to implement a school intervention model have an impact on 
outcomes for low-performing schools? 

Whether the type of model was related to improvements in student outcomes 

7. Was the type of school intervention model implemented related to improvement in 
outcomes for low-performing schools? 

Here we describe the study sample, the data collected, and the methods we used to analyze 
the data. The sample for the SIG evaluation included 22 states and approximately 60 districts. 
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Data on the use of SIG-promoted practices came from surveys of approximately 480 school 
administrators conducted in spring 2012 and spring 2013. Data on student outcomes came from 
student-level administrative data obtained from states and districts. We analyzed the 
implementation of SIG through both a descriptive analysis (that compared survey responses from 
290 schools that implemented SIG-funded models in 2012-2013 to 190 schools that did not) and 
a more rigorous RDD analysis (that compared survey responses from 190 schools eligible for 
SIG funds in 2010-2011 to 270 schools that were not eligible for SIG). We examined the overall 
impact of implementing any of the four SIG-funded models using an RDD analysis that 
compared test score data from the 190 schools that met the SIG eligibility criteria to the 270 
schools that missed the cutoff for eligibility. We used a descriptive, correlational analysis of 270 
schools that implemented a SIG-funded model in 2010-2011 to examine whether certain 
intervention models appeared more effective than others at improving student achievement. 

Prior to receiving a grant, SIG-funded model schools had baseline 
characteristics similar to those of other study schools 

Interpreting the differences between schools that implemented a SIG-funded model and 
schools that did not requires understanding the characteristics of these two groups of schools at 
baseline (during the 2009-2010 school year, which was prior to SIG funding receipt). 

• Schools that implemented a SIG-funded model and schools that did not had similar 
observable characteristics prior to receipt of the 2010 SIG awards. 2 These included 
several student and school demographic characteristics, such as race/ethnicity, percentage of 
students eligible for free or reduced-price lunch, Title I eligibility, location, and school level 
(elementary, middle, high school). 

• Study schools implementing a SIG-funded model were generally not representative of 
all U.S. schools implementing such models. The schools in our study that were 
implementing SIG-funded models were more disadvantaged and more likely to be in an 
urban area than U.S. schools nationally that were implementing such models. In particular, 
study schools implementing a SIG-funded model had higher percentages of students eligible 
for free or reduced-price lunch than U.S. schools nationally implementing such models (83 
percent versus 77 percent), and were more likely to be located in an urban area (88 percent 
versus 58 percent). Because the SIG sample is not representative of schools nationwide, the 
findings here may not apply to all schools nationally. 

Detailed findings 

Schools implementing SIG-funded models reported using more SIG-promoted practices 
than other schools, but we found no evidence that SIG caused those schools to implement 
more practices 

We examined whether implementing a SIG-funded model was associated with using more 
SIG-promoted practices. This is an important first step in understanding the extent to which the 
SIG program might improve student achievement. If schools implementing a SIG-funded model 


2 

“ The baseline characteristics examined came from the Common Core of Data. To limit respondent burden, the 
school administrator surveys focused primarily on practices schools were using in spring 2012 and spring 2013. 
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used the same practices as similar schools that did not implement a SIG-funded model, it is less 
likely that any changes in outcomes for SIG schools — positive or negative — could be attributed 
to the program. 

Using a descriptive analysis, we found that in spring 2013, schools that implemented a SIG- 
funded model reported using more practices, on average, than schools that did not in the 
following areas (Figure ES.l): 

• Comprehensive instructional reform strategies (90 percent of practices reported by SIG- 
funded model schools compared to 87 percent for schools not implementing such models) 

• Teacher and principal effectiveness (57 percent of practices reported by SIG-funded model 
schools compared to 49 percent for schools not implementing such models) 

• Learning time and community-oriented schools (68 percent of practices reported by SIG- 
funded model schools compared to 59 percent for schools not implementing such models) 

• Operational flexibility and support (47 percent of practices reported by SIG-funded model 
schools compared to 38 percent for schools not implementing such models) 

Adding up the differences across the four areas, schools implementing a SIG-funded model 
reported using more SIG-promoted practices overall (65 percent of the 35 practices examined, or 
22.8 practices) than schools not implementing one (58 percent of the 35 practices examined, or 
20.3 practices), a difference of 7 percentage points (2.5 practices). It is not clear whether a 
difference of this size would be meaningful in its overall influence on improvement practices and 
school outcomes. 

The spring 2013 findings presented in this report were generally the same as the spring 2012 
findings presented in an earlier report from this evaluation (Dragoset et al. 2015). 
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Figure ES.1. Use of practices promoted by SIG, by topic area 



Comprehensive 
Instructional Reform 
Strategies 
(8 items) 


Teacher and 
Principal 
Effectiveness 
(20 items) 


Learning Time and 
Community-Oriented 
Schools 
(5 items) 


Operational 
Flexibility 
and Support 
(2 items) 


■ Schools Implementing a SIG-Funded Intervention Model in 2012-2013 (N=290) 

Schools Not Implementing a SIG-Funded Intervention Model in 2012-2013 (N=190) 

Source: Surveys of school administrators in spring 2013. 

Note: The total number of practices (shown in parentheses below each bar) differed by topic area. This figure 

reads as follows (using the first bar on the left as an example): schools implementing a SIG-funded model 
reported using 90 percent of the practices in the comprehensive instructional reform strategies area. 
‘Significantly different from schools not implementing a SIG-funded model in 2012-2013 at the 0.05 level, two-tailed 
test. 


In addition to the descriptive analysis just described, we also used a more rigorous RDD 
analysis to examine whether implementing a SIG-funded model had an impact on schools’ use of 
practices. For schools near the SIG eligibility cutoff, we found that the implementation of a SIG- 
funded model had no significant impact on the total number of SIG-promoted practices used by 
schools in either spring 2012 or spring 2013 (Figure ES.2). The differences between schools that 
just met the SIG eligibility criteria and those that just missed the criteria were 0.4 practices (1 
percentage point) in spring 2012 and 3.3 practices (9 percentage points) in spring 2013. Although 
these differences were similar in size to the differences we observed in the descriptive analysis 
(particularly for 2013), they were not statistically significant. One likely reason why these 
differences were statistically significant in the descriptive analysis but not in the RDD analysis is 
that the RDD analysis was less able than the descriptive analysis to detect differences in the 
number of practices used. 3 Therefore, although our analyses show that schools implementing 

The minimum detectable differences for the RDD analysis were 5.8 practices in spring 2012 and 5.2 practices in 
spring 2013. In contrast, the minimum detectable difference was 0.8 practices in spring 2013 for the descriptive 
analysis presented in this report. 
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SIG-funded models used more SIG-promoted practices than other schools, we are unable to 
conclude that SIG caused those observed differences. 

Figure ES.2. Impacts of SIG-funded models on number of SIG-promoted 
practices used 

30- 

20 - 


cu 10- 



- 20 - 

-30- 

Spring 2012 Spring 2013 

Source: State and district administrative records; surveys of school administrators in spring 2012 and 2013. 

Note: Units are the number of practices used, out of 35 practices examined. Black lines show 95 percent 

confidence intervals. This figure reads as follows (using the first bar on the left as an example): in spring 
2012, schools that implemented a SIG-funded model used 0.4 more practices than schools that did not 
implement such a model, but this difference was not statistically significant. The results shown in this figure 
were calculated using the RDD methods described in Chapter II and Appendix A. 

Across all study schools, use of SIG-promoted practices was highest in the comprehensive 
instructional reform strategies area and lowest in the operational flexibility and support 
area 

Use of SIG-promoted practices was highest in the comprehensive instructional reform 
strategies area, in which schools reported using, on average, 7.1 of the 8 SIG-promoted practices 
examined (89 percent). Use of SIG-promoted policies and practices was lowest in the operational 
flexibility and support area. In that area, schools reported using, on average, 0.87 of the 2 SIG- 
promoted practices examined (43 percent). 

Across all topic areas, the use of individual practices varied widely. Nearly all study schools 
reported using benchmark or interim assessments at least once per year (a practice in the 
comprehensive instructional refonn strategies area). In contrast, very few study schools reported 
(1) using teacher evaluation results to infonn decisions about compensation, (2) using principal 
evaluation results to inform decisions about compensation, or (3) using financial incentives to 
recruit and retain effective principals (practices in the teacher and principal effectiveness topic 
area). 


ES-9 



EXECUTIVE SUMMARY 


SCHOOL IMPROVEMENT GRANTS: IMPLEMENTATION AND EFFECTIVENESS 


In three of four areas, changes over time in use of SIG-promoted practices did not 
significantly differ between schools implementing a SIG-funded model and schools not 
implementing one 

In three areas — comprehensive instructional reform strategies, teacher and principal 
effectiveness, and operational flexibility and support — there were no differences between the two 
groups of schools with respect to changes over time in practices used. In the fourth area — 
learning time and community-oriented schools — the schools implementing a SIG-funded model 
reported a decrease of 14 percent of practices between 2011-2012 and 2012-2013, compared to 
a decrease of 4 percent for schools not implementing such a model. 

There were no significant differences in use of ELL-focused practices promoted by SIG 
between schools implementing a SIG-funded model and schools not implementing one 

Both groups of schools (those implementing a SIG-funded model and those not 
implementing one) reported using 52 percent of the ELL-focused practices examined. 

Use of ELL-focused practices did not differ based on the prevalence of ELL students in the 
school, but SIG-funded model schools with higher ELL achievement gaps used these 
practices more than schools with lower gaps 

The differences in use of ELL-focused practices between schools with higher and lower 
ELL populations were not significant (0.4 practices among schools implementing a SIG-funded 
model and 0.3 practices among schools not implementing one). However, among schools 
implementing a SIG-funded model, schools with higher ELL achievement gaps reported using 
significantly more ELL-focused practices than schools with lower ELL achievement gaps (0.3 
more practices). 

SIG-funded models had no significant impact on test scores, high school graduation, or 
college enrollment 

We found no effect of SIG-funded models on student outcomes for schools near the SIG 
eligibility cutoff. When we examined the impacts of SIG-funded models on math and reading 
test scores, high school graduation, and college enrollment for 2010-2011, 2011-2012, and 
2012-2013, we found no significant impacts (Ligure ES.3 shows results for 2012-2013; 
Appendix A presents results for earlier years [2010-2011 and 2011-2012]). Lor 2012-2013, the 
impact on math test scores was 0.01 standard deviations, the impact on reading test scores was 
0.08 standard deviations, and the impact on high school graduation was -5 percentage points. We 
were unable to calculate an impact on college enrollment for 2012-2013 due to insufficient 
sample sizes, but we found no significant impacts on college enrollment for the other two school 
years (the impacts for 2010-2011 and 2011-2012 were -11 and 2 percentage points). Lor all of 
these student outcomes, we found no significant impacts within student and school subgroups. 
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Figure ES.3. Impacts of SIG-funded models on student outcomes 



| Math | Reading | Graduation*' 

Source: State and district administrative records. 

Note: Units for test scores are effect sizes (test scores were standardized to have a standard deviation of 1 ). 

Units for high school graduation are percentage points/100. For example, an impact of 0.1 indicates an 
increase of 10 percentage points. Black bars show 95 percent confidence intervals. The results shown in 
this figure were calculated using the RDD methods described in Chapter II and Appendix A. 

In elementary grades, there was no evidence that one model was associated with larger 
student achievement gains than another 

For elementary grades (2nd through 5th), we found no evidence that one intervention model 
was associated with larger student achievement gains than another. Between 2009-2010 (the 
year prior to SIG implementation) and 2012-2013, there were no significant differences in math 
or reading gains between schools implementing different models (Figure ES.4 presents math 
results; see Appendix B for reading results). This finding was also true for the two other outcome 
years we examined (2010-2011 and 2011-2012) and across all sensitivity analyses (see 
Appendix B). 
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Figure ES.4. Changes in math test scores in elementary grades, by model 



Source: State administrative data. 

Notes: This figure depicts regression-adjusted changes in math test scores between the baseline year (2009- 

2010) and 2012-2013 in grades 2 through 5. Changes in math test scores were regression-adjusted for 
state and grade using a linear model. The key finding (that no model was associated with larger student 
achievement gains than another) remained the same when we calculated changes in math test scores in a 
way that accounted for student mobility. Units are normal curve equivalents (NCEs). The NCEs reported in 
this figure correspond to the following effect sizes (ESs): transformation ES = 0.18, turnaround ES = 0.17, 
restart ES = 0.25. There were no statistically significant differences between schools implementing different 
models. 


In higher grades, the turnaround model was associated with larger student achievement 
gains in math than the transformation model 

For higher grades (6th through 12th), the implementation of the turnaround model was 
associated with larger student achievement gains than the transformation model. In particular, 
between 2009-2010 and 2012-2013, turnaround schools experienced larger gains in math than 
transformation schools (Figure ES.5 shows math results; see Appendix B for reading results). 

However, factors other than the SIG model implemented, such as baseline differences 
between schools implementing different models, may explain these differences in achievement 
gains. In particular, turnaround schools served more economically disadvantaged and lower- 
achieving students at baseline than transformation schools. This finding suggests that turnaround 
schools may have been fundamentally different from transformation schools prior to SIG, 
meaning that any number of explanations (other than the model implemented) could account for 
the different achievement gains. 
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Figure ES.5. Changes in math test scores in higher grades, by model 
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Source: State administrative data. 

Notes: This figure depicts regression-adjusted changes in math test scores between the baseline year (2009- 

2010) and 2012-2013 in grades 6 through 12, using changes calculated in a way that accounted for 
student mobility. Changes in math test scores were regression-adjusted for state and grade using a linear 
model. When we calculated changes in math test scores in a way that did not account for student mobility, 
we found that both the turnaround and restart models were associated with larger student achievement 
gains than the transformation model. Units are normal curve equivalents (NCEs). The NCEs reported in this 
figure correspond to the following effect sizes (ESs): transformation ES = 0.08, turnaround ES = 0.28, 
restart ES = 0.19. 

‘Significantly different from transformation model. 


Conclusions 

The findings in this report suggest that the SIG program did not have an impact on the use of 
practices promoted by the program or on student outcomes (including math or reading test 
scores, high school graduation, or college enrollment), at least for schools near the SIG eligibility 
cutoff. In higher grades (6th through 12th), the turnaround model was associated with larger 
student achievement gains in math than the transformation model. However, factors other than 
the SIG model implemented, such as unobserved differences between schools implementing 
different models, may explain these differences in achievement gains. 

These findings have broader relevance beyond the SIG program. In particular, the school 
improvement practices promoted by SIG were also promoted in the Race to the Top program. In 
addition, some of the SIG-promoted practices focused on teacher evaluation and compensation 
policies that were also a focus of Teacher Incentive Fund grants. All three of these programs 
involved large investments to support the use of practices with the goal of improving student 
outcomes. The findings presented in this report do not lend much support for the SIG program 


ES-13 


EXECUTIVE SUMMARY 


SCHOOL IMPROVEMENT GRANTS: IMPLEMENTATION AND EFFECTIVENESS 


having achieved this goal, as the program did not appear to have had an impact on the practices 
used by schools or on student outcomes, at least for schools near the SIG eligibility cutoff. 

However, it is important to keep in mind that our impact estimates only apply to schools 
near the SIG eligibility cutoff. They correspond to what might be expected if a policy change 
slightly shifted the cutoff for SIG eligibility, slightly increasing or decreasing the number of 
schools eligible for SIG funds. We cannot say whether SIG had an impact on use of practices or 
student outcomes for schools far away from the cutoff. 


ES-14 





